Systems and methods for a digital image sensor

ABSTRACT

A system, method, and computer program product for generating an image stack, comprising initializing a pixel array configured to include a set of analog storage planes, enabling simultaneous integration of the photographic scene for two or more analog storage planes within the set of analog storage planes, enabling integration to proceed during a first sampling interval, disabling integration for at least one analog storage plane, and enabling integration to proceed during a second sampling interval.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 61/852,427, titled “DIGITAL IMAGE SENSOR,” filed Mar. 15, 2013,which is hereby incorporated by reference for all purposes.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to digitalphotographic systems, and more specifically to systems and methods for adigital image sensor.

BACKGROUND

A typical digital camera focuses an optical image of a scene onto animage sensor, which samples the optical image to generate an electronicrepresentation of the scene. The electronic representation is thenprocessed and stored as a digital photograph. A conventional imagesensor is configured to generate a two-dimensional array of color pixelsfrom the optical image. Each color pixel typically includes anindependent intensity value for standard red, green, and bluewavelengths. A properly generated digital photograph will have a naturalappearance, resembling direct observation of the scene by a humanobserver. To generate digital photographs having a natural appearance,digital cameras attempt to mimic certain aspects of human visualperception.

One aspect of human visual perception that digital cameras mimic isdynamic adjustment to scene intensity. The human eye is able to adjustto a wide range of light intensity in the same scene. Digital camerasdynamically adjust to scene intensity by selecting a shutter speed,sampling sensitivity (“ISO” sensitivity associated with sensorsensitivity), and aperture to yield a good overall exposure level whengenerating the digital photograph. However, for a given exposuresetting, a typical scene may include areas spanning a dynamic range thatexceeds the dynamic range of a conventional image sensor, leading tooverexposure, underexposure, or a combination of both in the same scene.The scene may also include important visual detail at intensities thatare poorly quantized by the image sensor when configured for a specificexposure level, leading to quantization error, which appears as“banding” or unwanted “posterization.”

Techniques known in the art as high dynamic range (HDR) photographyprovide for sampling and representing image information having a highdynamic range substantially representative of dynamic range within agiven scene. HDR photography conventionally involves sampling a set ofdigital photographs, referred to as an image stack, at differentexposure levels for the same scene to capture image detail at differentdynamic range levels. Images comprising an image stack may be combinedto synthesize a single digital photograph that represents contrast andimage detail depicting the full dynamic range of the scene. In certainscenarios, the full dynamic range is mapped to a reduced dynamic rangefor display on a conventional display device, such as a liquid crystaldisplay (LCD). The digital photographs comprising the image stack areassumed to contain substantially consistent content that is sampled atdifferent exposures. The digital photographs are conventionally sampledsequentially, with an inter-sample time separating the capture of eachdigital photograph.

During sequential sampling, the digital camera may move, such as fromhand motion or vibration. During sequential sampling, the scene may alsochange, such as from people, animals, or objects moving in the scene. Asa consequence of such motion or change, each digital photograph withinthe set of digital photographs needs to be aligned to the other digitalphotographs to provide spatial consistency prior to a combinationoperation. As inter-sample time increases, the likelihood ofuncorrectable misalignment among the digital photographs increases, asdoes the likelihood of uncorrectable divergent content within thedigital photographs. Examples of divergent content include birds flyingin a landscape scene, and people talking or otherwise moving in a socialscene. A common example of uncorrectable divergent content arises when aperson is moving their head, such that the first digital photograph inthe set of digital photographs captures the person's face, while asecond digital photograph captures the side of the person's head.Conventional alignment techniques are computationally intense and stillcannot adequately correct for content changes, such as aligning a facewith the side of a head. Furthermore, conventional image capturetechniques typically require significant inter-sample time, which may begreater than a thirtieth of one second. Significant inter-sample timecommonly leads to capturing uncorrectable divergent content or digitalphotographs that cannot be properly aligned within the image stack,thereby visibly and negatively impacting the quality of syntheticdigital photographs generated from the image stack.

As the foregoing illustrates, there is a need for addressing this and/orother related issues associated with the prior art.

SUMMARY

A system, method, and computer program product for generating an imagestack, comprising initializing a pixel array configured to include a setof analog storage planes, enabling simultaneous integration of thephotographic scene for two or more analog storage planes within the setof analog storage planes, enabling integration to proceed during a firstsampling interval, disabling integration for at least one analog storageplane, and enabling integration to proceed during a second samplinginterval.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1A illustrates a flow chart of a method for generating an imagestack comprising two or more images of a photographic scene, inaccordance with one embodiment;

FIG. 1B illustrates a flow chart of a method for generating an imagestack comprising an ambient image and a strobe image of a photographicscene, in accordance with one embodiment;

FIG. 2 illustrates generating a synthetic image from an image stack,according to one embodiment of the present invention;

FIG. 3A illustrates a digital camera, configured to implement one ormore aspects of the present invention;

FIG. 3B illustrates a mobile device, configured to implement one or moreaspects of the present invention;

FIG. 3C illustrates a digital photographic system, configured toimplement one or more aspects of the present invention;

FIG. 3D illustrates a processor complex within the digital photographicsystem, in accordance with one embodiment;

FIG. 3E illustrates a camera module configured to control a strobe unitthrough a strobe control signal, according to one embodiment of thepresent invention;

FIG. 3F illustrates a camera module configured to sample an image basedon state information for a strobe unit, according to one embodiment ofthe present invention;

FIG. 4A illustrates a block diagram of image sensor, according to oneembodiment of the present invention;

FIG. 4B illustrates a pixel array within an image sensor, according toone embodiment of the present invention;

FIG. 4C illustrates a color filter configuration comprising red, green,and blue filters for one pixel within a pixel array, according to oneembodiment of the present invention;

FIG. 4D illustrates a color filter configuration comprising red, green,blue, and white filters for one pixel within a pixel array, according toone embodiment of the present invention;

FIG. 4E illustrates a color filter configuration comprising cyan,magenta, yellow, and white filters for one pixel within a pixel array,according to one embodiment of the present invention;

FIG. 4F illustrates a cross-section of color cells within a pixel array,according to one embodiment of the present invention;

FIG. 5 is a circuit diagram for a photo-sensitive cell within a pixelimplemented using complementary-symmetry metal-oxide semiconductordevices, according;

FIG. 6A is a circuit diagram for a first photo-sensitive cell, accordingto one embodiment;

FIG. 6B is a circuit diagram for a second photo-sensitive cell,according to one embodiment;

FIG. 6C is a circuit diagram for a third photo-sensitive cell, accordingto one embodiment;

FIG. 6D depicts exemplary physical layout for a pixel comprising fourphoto-sensitive cells, according to one embodiment;

FIG. 7A illustrates exemplary timing for controlling cells within apixel array to sequentially capture an ambient image and a strobe imageilluminated by a strobe unit, according to one embodiment of the presentinvention;

FIG. 7B illustrates exemplary timing for controlling cells within apixel array to concurrently capture an ambient image and an imageilluminated by a strobe unit, according to one embodiment of the presentinvention;

FIG. 7C illustrates exemplary timing for controlling cells within apixel array to concurrently capture two ambient images having differentexposures, according to one embodiment of the present invention;

FIG. 7D illustrates exemplary timing for controlling cells within apixel array to concurrently capture two ambient images having differentexposures, according to one embodiment of the present invention;

FIG. 7E illustrates exemplary timing for controlling cells within apixel array to concurrently capture four ambient images, each havingdifferent exposure times, according to one embodiment of the presentinvention; and

FIG. 7F illustrates exemplary timing for controlling cells within apixel array to concurrently capture three ambient images havingdifferent exposures and subsequently capture a strobe image, accordingto one embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention enable a digital photographicsystem to capture an image stack for a photographic scene. Exemplarydigital photographic systems include, without limitation, digitalcameras and mobile devices such as smart phones that are configured toinclude a digital camera module. A given photographic scene is a portionof an overall scene sampled by the digital photographic system. Two ormore images are sampled by the digital photographic system to generatean image stack.

A given image stack comprises images of the photographic scene sampledwith potentially different exposure, different strobe illumination, or acombination thereof. For example, each image within the image stack maybe sampled according to a different exposure time, exposure sensitivity,or a combination thereof. A given image within the image stack may besampled in conjunction with or without strobe illumination added to thephotographic scene. Images comprising an image stack should be sampledover an appropriately short span of time to reduce visible differencesor changes in scene content among the images. In one embodiment, imagescomprising a complete image stack are sampled within one second. Inanother embodiment, images comprising a complete image stack are sampledwithin a tenth of a second.

In one embodiment, two or more images are captured according todifferent exposure levels during overlapping time intervals, therebyreducing potential changes in scene content among the two or moreimages. In other embodiments, the two or more images are sampledsequentially under control of an image sensor circuit to reduceinter-image time. In certain embodiments, at least one image of the twoor more images is sampled in conjunction with a strobe unit beingenabled to illuminate a photographic scene. Image sampling may becontrolled by the image sensor circuit to reduce inter-image timebetween an image sampled using only ambient illumination and an imagesampled in conjunction with strobe illumination. The strobe unit maycomprise a light-emitting diode (LED) configured to illuminate thephotographic scene.

In one embodiment, each pixel of an image sensor comprises a set ofphoto-sensitive cells, each having specific color sensitivity. Forexample, a pixel may include a photo-sensitive cell configured to besensitive to red light, a photo-sensitive cell configured to besensitive to blue light, and two photo-sensitive cells configured to besensitive to green light. Each photo-sensitive cell is configured toinclude two or more analog sampling circuits. A set of analog samplingcircuits comprising one analog sampling circuit per photo-sensitive cellwithin the image sensor may be configured to sample and store a firstimage. Collectively, one set of analog sampling circuits forms acomplete image plane and is referred to herein as an analog storageplane. A second set of substantially identically defined analog samplingcircuits within the image sensor may be configured to sample and store asecond image. A third set of substantially identically defined storageelements within the image sensor may be configured to sample and store athird image, and so forth. Hence an image sensor may be configured tosample and simultaneously store multiple images within analog storageplanes.

Each analog sampling circuit may be independently coupled to aphotodiode within the photo-sensitive cell, and independently read. Inone embodiment, the first set of analog sampling circuits are coupled tocorresponding photodiodes for a first time interval to sample a firstimage having a first corresponding exposure time. A second set of analogsampling circuits are coupled to the corresponding photodiodes for asecond time interval to sample a second image having a secondcorresponding exposure time. In certain embodiments, the first timeduration overlaps the second time duration, so that the first set ofanalog sampling circuits and the second set of analog sampling circuitsare coupled to the photodiode concurrently during an overlap time. Inone embodiment, the overlap time is within the first time duration.Current generated by the photodiode is split over the number of analogsampling circuits coupled to the photodiode at any given time.Consequently, exposure sensitivity varies as a function how many analogsampling circuits are coupled to the photodiode at any given time andhow much capacitance is associated with each analog sampling circuit.Such variation needs to be accounted for in determining exposure timefor each image.

FIG. 1A illustrates a flow chart of a method 100 for generating an imagestack comprising two or more images of a photographic scene, inaccordance with one embodiment. Although method 100 is described inconjunction with the systems of FIGS. 2A-3B, persons of ordinary skillin the art will understand that any system that performs method 100 iswithin the scope and spirit of embodiments of the present invention. Inone embodiment, a digital photographic system, such as digitalphotographic system 300 of FIG. 3A, is configured to perform method 100.The digital photographic system may be implemented within a digitalcamera, such as digital camera 202 of FIG. 2A, or a mobile device, suchas mobile device 204 of FIG. 2B. In certain embodiments, a cameramodule, such as camera module 330 of FIG. 3C, is configured to performmethod 100. Method 100 may be performed with or without a strobe unit,such as strobe unit 336, enabled to contribute illumination to thephotographic scene.

Method 100 begins in step 110, where the camera module configuresexposure parameters for an image stack to be sampled by the cameramodule. Configuring the exposure parameters may include, withoutlimitation, writing registers within an image sensor comprising thecamera module that specify exposure time for each participating analogstorage plane, exposure sensitivity for one or more analog storageplanes, or a combination thereof. Exposure parameters may be determinedprior to this step according to any technically feasible technique, suchas well-known techniques for estimating exposure based on measuringexposure associated with a sequence of test images sampled usingdifferent exposure parameters.

In step 112, the camera module receives a capture command. The capturecommand directs the camera module to sample two or more imagescomprising the image stack. The capture command may result from a userpressing a shutter release button, such as a physical button or a userinterface button. In step 114, the camera module initializes a pixelarray within the image sensor. In one embodiment, initializing the pixelarray comprises driving voltages on internal nodes of photo-sensitivecells within one or more analog storage planes to a reference voltage,such as a supply voltage or a bias voltage. In step 116, the cameramodule enables analog sampling circuits within two or more analogstorage planes to simultaneously integrate (accumulate) an imagecorresponding to a photographic scene. In one embodiment, integrating animage comprises each analog sampling circuit within an analog storageplane integrating a current generated by a corresponding photodiode. Instep 118, analog sampling circuits within enabled analog storage planesintegrate a respective image during a sampling interval. Each samplinginterval may comprise a different time duration:

If, in step 120, the camera module should sample another image, then themethod proceeds to step 122, where the camera module disables samplingfor one analog storage plane within the image sensor. Upon disablingsampling for a given analog storage plane, an image associated with theanalog storage plane has been sampled completely for an appropriateexposure time.

Returning to step 120, if the camera module should not sample anotherimage then the method terminates. The camera module should not sampleanother image after the last sampling interval has lapsed and samplingof the last image has been completed.

Reading an image from a corresponding analog storage plane may proceedusing any technically feasible technique.

FIG. 1B illustrates a flow chart of a method 102 for generating an imagestack comprising an ambient image and a strobe image of a photographicscene, in accordance with one embodiment. Although method 102 isdescribed in conjunction with the systems of FIGS. 2A-3B, persons ofordinary skill in the art will understand that any system that performsmethod 102 is within the scope and spirit of embodiments of the presentinvention. In one embodiment, a digital photographic system, such asdigital photographic system 300 of FIG. 3A, is configured to performmethod 102. The digital photographic system may be implemented within adigital camera, such as digital camera 202 of FIG. 2A, or a mobiledevice, such as mobile device 204 of FIG. 2B.

Method 102 begins in step 140, where the camera module configuresexposure parameters for an image stack to be sampled by the cameramodule. Configuring the exposure parameters may include, withoutlimitation, writing registers within an image sensor comprising thecamera module that specify exposure time for each participating analogstorage plane, exposure sensitivity for one or more analog storageplanes, or a combination thereof. Exposure parameters may be determinedprior to this step according to any technically feasible technique, suchas well-known techniques for estimating exposure based on measuringexposure associated with a sequence of test images sampled usingdifferent exposure parameters.

In step 142, the camera module receives a capture command. The capturecommand directs the camera module to sample two or more imagescomprising the image stack. The capture command may result from a userpressing a shutter release button, such as a physical button or a userinterface button. In step 144, the camera module initializes a pixelarray within the image sensor. In one embodiment, initializing the pixelarray comprises driving voltages on internal nodes of photo-sensitivecells within one or more analog storage planes to a reference voltage,such as a supply voltage or a bias voltage.

In step 146, the camera module samples one or more ambient images withincorresponding analog storage planes. In one embodiment, step 146implements steps 116 through 122 of method 100 of FIG. 1A.

In step 150, the camera module determines that a strobe unit, such asstrobe unit 336 of FIG. 3C, is enabled. In one embodiment, determiningthat the strobe unit is enabled includes the camera module directlyenabling the strobe unit, such by transmitting a strobe control commandthrough strobe control signal 338. In another embodiment, determiningthat the strobe unit is enabled includes the camera module detectingthat the strobe unit has been enabled, such as by processor complex 310.

In step 152, the camera module samples one or more strobe images withincorresponding analog storage planes. In one embodiment, step 152implements steps 116 through 122 of method 100 of FIG. 1A. In oneembodiment, the camera module directly disables the strobe unit aftercompleting step 152, such as by transmitting a strobe control commandthrough strobe control signal 338. In another embodiment, processorcomplex 310 disables the strobe unit after the camera module completesstep 152.

In certain embodiments, the camera module is configured to store bothambient images and strobe images concurrently within analog storageplanes. In other embodiments, the camera module offloads one or moreambient images prior to sampling a strobe image.

FIG. 2 illustrates generating a synthetic image 250 from an image stack200, according to one embodiment of the present invention. As shown,image stack 200 includes images 210, 212, and 214 of a photographicscene comprising a high brightness region 220 and a low brightnessregion 222. In this example, image 212 is exposed according to overallscene brightness, thereby generally capturing scene detail. Image 212may also potentially capture some detail within high brightness region220 and some detail within low brightness region 222. Image 210 isexposed to capture image detail within high brightness region 220. Forexample, image 210 may be exposed according to an exposure offset (e.g.,one or more exposure stops down) relative to image 212. Alternatively,image 210 may be exposed according to local intensity conditions for oneor more of the brightest regions in the scene. For example, image 210may be exposed according to high brightness region 220, to the exclusionof other regions in the scene having lower overall brightness.Similarly, image 214 is exposed to capture image detail within lowbrightness region 222. To capture low brightness detail within thescene, image 214 may be exposed according to an exposure offset (e.g.,one or more exposure stops up) relative to image 212. Alternatively,image 214 may be exposed according to local intensity conditions for oneor more of the darker regions of the scene.

An image blend operation 240 generates synthetic image 250 from imagestack 200. As depicted here, synthetic image 250 includes overall imagedetail, as well as image detail from high brightness region 220 and lowbrightness region 222. Image blend operation 240 may implement anytechnically feasible operation for blending an image stack. For example,any high dynamic range (HDR) blending technique may be implemented toperform image blend operation 240. Exemplary blending techniques knownin the art include bilateral filtering, global range compression andblending, local range compression and blending, and the like.

To properly perform a blend operation, Images 210, 212, 214 need to bealigned to so that visible detail in each image is positioned in thesame location in each image. For example, feature 225 in each imageshould be located in the same position for the purpose of blendingimages 210, 212, 214 to generate synthetic image 250. Misalignment canresult in blurring or ghosting in synthetic image 250. Varioustechniques are known in the art for aligning images that may have beentaken from a slightly different camera position. However, if scenecontent changes, then alignment may fail, leading to a poor qualitysynthetic image 250. Scene content may change in a conventional camerasystem because inter-sample time between images 210, 212, and 214 issufficiently long so as to capture discernible movement of subjectscomprising scene content for images comprising an image stack. In manytypical scenarios, ten milliseconds or more of inter-sample time issufficiently long to result in discernible movement of commonphotographic subject matter. Furthermore, in certain scenarios, camerashake introduces discernible blur into synthetic image 250.

Embodiments of the present invention serve to reduce or eliminateinter-sample time for two or more images comprising an image stack. Inone embodiment, an image stack is captured by a digital photographicsystem, described below in greater detail. A strobe unit may be enabledto provide illumination in conjunction with sampling one or more imageswithin the image stack.

FIG. 3A illustrates a digital camera 302, configured to implement one ormore aspects of the present invention. Digital camera 302 includes adigital photographic system, such as digital photographic system 300 ofFIG. 3C, configured to generate an image stack by sampling aphotographic scene as described in conjunction with method 100 of FIG.1A or method 102 of FIG. 1B. A camera module 330 is configured to sampleimages comprising the image stack.

Digital camera 302 may include a strobe unit 336, and may include ashutter release button 315 for triggering a photographic sample event,whereby digital camera 302 samples two or more images comprising animage stack. Any other technically feasible shutter release command maytrigger the photographic sample event, such as a timer trigger or remotecontrol receiver configured to generate a shutter release command.

Embodiments of the present invention advantageously enable the cameramodule 330 to sample images comprising the image stack with lowerinter-sample time than conventional techniques. In certain embodiments,the images are sampled during overlapping time intervals, which mayreduce the inter-sample time to zero. In other embodiments, the cameramodule 330 samples images in coordination with the strobe unit 336 toreduce inter-sample time between an image sampled without strobeillumination and an image sampled subsequently with strobe illumination.

FIG. 3B illustrates a mobile device 304, configured to implement one ormore aspects of the present invention. Mobile device 304 includes adigital photographic system, such as digital photographic system 300 ofFIG. 3C, configured to generate an image stack by sampling aphotographic scene as described in conjunction with method 100 of FIG.1A or method 102 of FIG. 1B. Camera module 330 is configured to sampleimages comprising the image stack. A shutter release command may begenerated through a mechanical button or a virtual button, which may beactivated by a touch gesture on a touch entry display system 311 withinmobile device 304, or by any other technically feasible trigger.

In one embodiment, the touch entry display system 311 is disposed on theopposite side of mobile device 304 from camera module 330. In certainembodiments, the mobile device 304 includes a user-facing camera module331 and may include a user-facing strobe unit (not shown). Theuser-facing camera module 331 and user-facing strobe unit are configuredto sample an image stack in accordance with method 100 of FIG. 1A ormethod 102 of FIG. 1B.

The digital camera 302 and the mobile device 304 may each generate andstores a synthetic image based on an image stack sampled by cameramodule 330. In one embodiment, the image stack includes a set of relatedimages sampled under only ambient lighting. In another embodiment, theimage stack includes at least one image sampled with strobeillumination, such as from strobe unit 336.

FIG. 3C illustrates a digital photographic system 300, configured toimplement one or more aspects of the present invention. Digitalphotographic system 300 includes a processor complex 310 coupled to acamera module 330 and a strobe unit 336. Digital photographic system 300may also include, without limitation, a display unit 312, a set ofinput/output devices 314, non-volatile memory 316, volatile memory 318,a wireless unit 340, and sensor devices 342, each coupled to processorcomplex 310. In one embodiment, a power management subsystem 320 isconfigured to generate appropriate power supply voltages for eachelectrical load element within digital photographic system 300. Abattery 322 may be configured to supply electrical energy to powermanagement subsystem 320. Battery 322 may implement any technicallyfeasible energy storage system, including primary or rechargeablebattery technologies.

In one embodiment, strobe unit 336 is integrated into digitalphotographic system 300 and configured to provide strobe illumination350 during an image sample event performed by digital photographicsystem 300. In an alternative embodiment, strobe unit 336 is implementedas an independent device from digital photographic system 300 andconfigured to provide strobe illumination 350 during an image sampleevent performed by digital photographic system 300. Strobe unit 336 maycomprise one or more LED devices. In certain embodiments, two or morestrobe units are configured to synchronously generate strobeillumination in conjunction with sampling an image.

In one embodiment, strobe unit 336 is directed through a strobe controlsignal 338 to either emit strobe illumination 350 or not emit strobeillumination 350. The strobe control signal 338 may implement anytechnically feasible signal transmission protocol. Strobe control signal338 may indicate a strobe parameter, such as strobe intensity or strobecolor, for directing strobe unit 336 to generate a specified intensityand/or color of strobe illumination 350. Strobe control signal 338 maybe generated by processor complex 310, camera module 330, or by anyother technically feasible combination thereof. In one embodiment,strobe control signal 338 is generated by a camera interface unit 386within the processor complex 310 and transmitted via an interconnect 334to both the strobe unit 336 and the camera module 330. In anotherembodiment, strobe control signal 338 is generated by camera module 330and transmitted to strobe unit 336 via interconnect 334.

Optical scene information 352, which may include strobe illumination 350reflected from objects in the photographic scene, is focused as anoptical image onto an image sensor 332, within camera module 330. Imagesensor 332 generates an electronic representation of the optical image.The electronic representation comprises spatial color intensityinformation, which may include different color intensity samples, suchas for red, green, and blue light. The spatial color intensityinformation may also include samples for white light. The electronicrepresentation is transmitted to processor complex 310 via interconnect334, which may implement any technically feasible signal transmissionprotocol.

Input/output devices 314 may include, without limitation, a capacitivetouch input surface, a resistive tablet input surface, one or morebuttons, one or more knobs, light-emitting devices, light detectingdevices, sound emitting devices, sound detecting devices, or any othertechnically feasible device for receiving user input and converting theinput to electrical signals, or converting electrical signals into aphysical signal. In one embodiment, input/output devices 314 include acapacitive touch input surface coupled to display unit 312.

Non-volatile (NV) memory 316 is configured to store data when power isinterrupted. In one embodiment, NV memory 316 comprises one or moreflash memory devices. NV memory 316 comprises a non-transitorycomputer-readable medium, which may be configured to include programminginstructions for execution by one or more processing units withinprocessor complex 310. The programming instructions may implement,without limitation, an operating system (OS), UI modules, imageprocessing and storage modules, one or more modules for sampling animage stack through camera module 330, one or more modules forpresenting the image stack or synthetic image generated from the imagestack through display unit 312. The programming instructions may alsoimplement one or more modules for merging images or portions of imageswithin the image stack, aligning at least portions of each image withinthe image stack, or a combination thereof. In one embodiment, theprogramming instructions are configured to implement method 100 or FIG.1A, method 102 of FIG. 1B, or a combination thereof. One or more memorydevices comprising NV memory 316 may be packaged as a module configuredto be installed or removed by a user. In one embodiment, volatile memory318 comprises dynamic random access memory (DRAM) configured totemporarily store programming instructions, image data such as dataassociated with an image stack, and the like, accessed during the courseof normal operation of digital photographic system 300.

Sensor devices 342 may include, without limitation, an accelerometer todetect motion and/or orientation, an electronic gyroscope to detectmotion and/or orientation, a magnetic flux detector to detectorientation, a global positioning system (GPS) module to detectgeographic position, or any combination thereof.

Wireless unit 340 may include one or more digital radios configured tosend and receive digital data. In particular, wireless unit 340 mayimplement wireless standards known in the art as “WiFi” based onInstitute for Electrical and Electronics Engineers (IEEE) standard802.11, and may implement digital cellular telephony standards for datacommunication such as the well-known “3G” and “4G” suites of standards.Wireless unit 340 may further implement standards and protocols known inthe art as LTE (long term evolution). In one embodiment, digitalphotographic system 300 is configured to transmit one or more digitalphotographs, sampled according to techniques taught herein, to an onlineor “cloud-based” photographic media service via wireless unit 340. Theone or more digital photographs may reside within either NV memory 316or volatile memory 318. In such a scenario, a user may possesscredentials to access the online photographic media service and totransmit the one or more digital photographs for storage andpresentation by the online photographic media service. The credentialsmay be stored or generated within digital photographic system 300 priorto transmission of the digital photographs. The online photographicmedia service may comprise a social networking service, photographsharing service, or any other network-based service that providesstorage and transmission of digital photographs. In certain embodiments,one or more digital photographs are generated by the online photographicmedia service based on an image stack sampled according to techniquestaught herein. In such embodiments, a user may upload source imagescomprising an image stack for processing by the online photographicmedia service.

In one embodiment, digital photographic system 300 comprises a pluralityof camera modules 330. Such an embodiment may also include at least onestrobe unit 336 configured to illuminate a photographic scene, sampledas multiple views by the plurality of camera modules 330. The pluralityof camera modules 330 may be configured to sample a wide angle view(greater than forty-five degrees of sweep among cameras) to generate apanoramic photograph. The plurality of camera modules 330 may also beconfigured to sample two or more narrow angle views (less thanforty-five degrees of sweep among cameras) to generate a stereoscopicphotograph.

Display unit 312 is configured to display a two-dimensional array ofpixels to form an image for display. Display unit 312 may comprise aliquid-crystal display, an organic LED display, or any other technicallyfeasible type of display. In certain embodiments, display unit 312 isable to display a narrower dynamic range of image intensity values thana complete range of intensity values sampled over a set of two or moreimages comprising a given image stack. Here, images comprising the imagestack may be merged according to any technically feasible HDR blendingtechnique to generate a synthetic image for display within dynamic rangeconstraints of display unit 312. In one embodiment, the limited dynamicrange specifies an eight-bit per color channel binary representation ofcorresponding color intensities. In other embodiments, the limiteddynamic range specifies a twelve-bit per color channel binaryrepresentation.

FIG. 3D illustrates a processor complex 310 within digital photographicsystem 300 of FIG. 3C, according to one embodiment of the presentinvention. Processor complex 310 includes a processor subsystem 360 andmay include a memory subsystem 362. In one embodiment, processor complex310 comprises a system on a chip (SoC) device that implements processorsubsystem 360, and memory subsystem 362 comprises one or more DRAMdevices coupled to processor subsystem 360. In one implementation,processor complex 310 comprises a multi-chip module (MCM) encapsulatingthe SoC device and the one or more DRAM devices.

Processor subsystem 360 may include, without limitation, one or morecentral processing unit (CPU) cores 370, a memory interface 380,input/output interfaces unit 384, and a display interface unit 382, eachcoupled to an interconnect 374. The one or more CPU cores 370 may beconfigured to execute instructions residing within memory subsystem 362,volatile memory 318, NV memory 316, or any combination thereof. Each ofthe one or more CPU cores 370 may be configured to retrieve and storedata through interconnect 374 and memory interface 380. Each of the oneor more CPU cores 370 may include a data cache, and an instructioncache. Two or more CPU cores 370 may share a data cache, an instructioncache, or any combination thereof. In one embodiment, a cache hierarchyis implemented to provide each CPU core 370 with a private cache layer,and a shared cache layer.

Processor subsystem 360 may further include one or more graphicsprocessing unit (GPU) cores 372. Each GPU core 372 comprises a pluralityof multi-threaded execution units that may be programmed to implementgraphics acceleration functions. GPU cores 372 may be configured toexecute multiple thread programs according to well-known standards suchas OpenGL™, OpenCL™, CUDA™, and the like. In certain embodiments, atleast one GPU core 372 implements at least a portion of a motionestimation function, such as a well-known Harris detector or awell-known Hessian-Laplace detector. Such a motion estimation functionmay be used for aligning images or portions of images within the imagestack.

Interconnect 374 is configured to transmit data between and among memoryinterface 380, display interface unit 382, input/output interfaces unit384, CPU cores 370, and GPU cores 372. Interconnect 374 may implementone or more buses, one or more rings, a cross-bar, a mesh, or any othertechnically feasible data transmission structure or technique. Memoryinterface 380 is configured to couple memory subsystem 362 tointerconnect 374. Memory interface 380 may also couple NV memory 316,volatile memory 318, or any combination thereof to interconnect 374.Display interface unit 382 is configured to couple display unit 312 tointerconnect 374. Display interface unit 382 may implement certain framebuffer functions such as frame refresh. Alternatively, display unit 312may implement frame refresh. Input/output interfaces unit 384 may beconfigured to couple various input/output devices to interconnect 374.

In certain embodiments, camera module 330 is configured to storeexposure parameters for sampling each image in an image stack. Whendirected to sample an image stack, the camera module 330 samples theimages comprising the image stack according to the stored exposureparameters. A software module comprising programming instructionsexecuting within processor complex 310 may generate and store theexposure parameters prior to directing the camera module 330 to samplethe image stack.

In one embodiment, exposure parameters associated with images comprisingthe image stack are stored within a parameter data structure. The camerainterface unit 386 is configured to read exposure parameters from theparameter data structure and to transmit the exposure parameters to thecamera module 330 in preparation of sampling an image stack. After thecamera module 330 is configured according to the exposure parameters,the camera interface unit 386 directs the camera module 330 to sample animage stack. The data structure may be stored within the camerainterface unit 386, within a memory circuit within processor complex310, within volatile memory 318, within NV memory 316, or within anyother technically feasible memory circuit. A software module executingwithin processor complex 310 may generate and store the data structure.

In one embodiment, the camera interface unit 386 transmits exposureparameters and commands to camera module 330 through interconnect 334.In certain embodiments, the camera interface unit 386 is configured todirectly control the strobe unit 336 by transmitting control commands tothe strobe unit 336 through strobe control signal 338. By directlycontrolling both the camera module 330 and the strobe unit 336, thecamera interface unit 386 may cause the camera module 330 and the strobeunit 336 to perform their respective operations in precise timesynchronization. In one embodiment, precise time synchronization isdefined to be less than five hundred microseconds of event timing error.

In other embodiments, a software module executing within processorcomplex 310 directs the operation and synchronization of camera module330 and the strobe unit 336, with potentially reduced performance, suchas event timing error of more than one millisecond.

In one embodiment, camera interface unit 386 is configured to accumulatestatistics while receiving image data from the camera module 330. Inparticular, the camera interface unit 386 may accumulate exposurestatistics for a given image while receiving image data for the imagethrough interconnect 334. Exposure statistics may include an intensityhistogram, a count of over-exposed pixels, a counter of under-exposedpixels, an intensity-weighted sum of pixel intensity, or any combinationthereof. The camera interface unit 386 may present the exposurestatistics as memory-mapped storage locations within a physical orvirtual address space defined by a processor, such as a CPU core 370,within processor complex 310. In one embodiment, the exposure statisticsare mapped in a memory-mapped register space, which may be accessedthrough interconnect 334. In other embodiments, the exposure statisticsare transmitted in conjunction with transmitting pixel data for acaptured image. For example, the exposure statistics for a given imagemay be transmitted as in-line data, following transmission of pixelintensity data for the image. Exposure statistics may be cached withincamera interface 386, after being received from camera module 330.

In certain embodiments, camera interface unit 386 accumulates colorstatistics for estimating scene white-balance. Any technically feasiblecolor statistics may be accumulated for estimating white balance, suchas a sum of intensities for different color channels comprising red,green, and blue color channels. The sum of color channel intensities maythen be used to perform a white-balance color correction on anassociated image, according to a white-balance model such as agray-world white-balance model. In other embodiments, curve-fittingstatistics are accumulated for a linear or a quadratic curve fit usedfor implementing white-balance correction on an image. In oneembodiment, camera interface unit 386 accumulates spatial colorstatistics for performing color-matching between or among images, suchas between or among an ambient image and one or more images sampled withstrobe illumination. As with the exposure statistics, the colorstatistics may be presented as memory-mapped storage locations withinprocessor complex 310. In one embodiment, the color statistics aremapped in a memory-mapped register space, which may be accessed throughinterconnect 334. In other embodiments, the color statistics aretransmitted in conjunction with transmitting pixel data for a capturedimage. For example, the color statistics for a given image may betransmitted as in-line data, following transmission of pixel intensitydata for the image. Color statistics may be cached within camerainterface 386 after being received from camera module 330.

In one embodiment, camera module 330 transmits strobe control signal 338to strobe unit 336, enabling strobe unit 336 to generate illuminationwhile the camera module 330 is sampling an image. In another embodiment,camera module 330 samples an image illuminated by strobe unit 336 uponreceiving an indication from camera interface unit 386 that strobe unit336 is enabled. In yet another embodiment, camera module 330 samples animage illuminated by strobe unit 336 upon detecting strobe illuminationwithin a photographic scene via a rapid rise in scene illumination. Instill yet another embodiment, camera module 330 enables strobe unit 336to generate strobe illumination while sampling one image and disablesstrobe unit 336 while sampling a different image.

FIG. 3E illustrates camera module 330 configured to control strobe unit336 through strobe control signal 338, according to one embodiment ofthe present invention. Lens 390 is configured to focus optical sceneinformation 352 onto image sensor 332 to be sampled. As shown, imagesensor 332, within the camera module 330, is configured to generatestrobe control signal 338 to enable strobe unit 336 while sampling animage specified to include strobe illumination as part of an imageexposure specification. Furthermore, image sensor 332 is configured todisable strobe unit 336 through strobe control signal 338 when samplingan image specified to not include strobe illumination. Image sensor 332advantageously controls detailed timing of strobe unit 336 to reduceinter-sample time between one or more images sampled with strobe unit336 enabled and one or more images sampled with strobe unit 336disabled. For example, inter-sample time between one or more images maybe reduced to less than one microsecond. Image sensor 332 is thus ableto directly control sampling operations, including enabling anddisabling strobe unit 336, associated with generating an image stack,which may comprise at least one image sampled with strobe unit 336disabled, and at least one image sampled with strobe unit 336 eitherenabled or disabled. In one embodiment, data comprising the image stacksampled by image sensor 332 is transmitted via interconnect 334 tocamera interface unit 386 within processor complex 310. In certainembodiments, camera module 330 includes control circuitry (not shown),configured to generate strobe control signal 338 in conjunction withcontrolling operation of the image sensor 332.

FIG. 3F illustrates a camera module 330 configured to sample an imagebased on state information for strobe unit 336, according to oneembodiment of the present invention. Commands for configuring stateinformation associated with the strobe unit 336 are transmitted throughstrobe control signal 338, which may be sampled by camera module 330 todetect when strobe unit 336 is enabled with relatively fine timeresolution. For example, camera module 330 may detect when strobe unit336 is enabled or disable within a microsecond or less of being enabledor disabled. To sample an image requiring strobe illumination, camerainterface unit 386 enables strobe unit 336 by sending an enable commandthrough strobe control signal 338. The enable command may comprise asignal level transition, a data packet, a register write, or any othertechnically feasible transmission of a command. Camera module 330 sensesthat strobe unit 336 is enabled and may cause image sensor 332 to sampleone or more images requiring strobe illumination while strobe unit 336is enabled.

FIG. 4A illustrates a block diagram of image sensor 332, according toone embodiment of the present invention. As shown, image sensor 332comprises row logic 412, a control (CTRL) unit 414, a pixel array 410, acolumn read out circuit 420, an analog-to-digital unit 422, and aninput/output interface unit 426. The image sensor 332 may also include astatistics unit 416.

Pixel array 410 comprises a two-dimensional array of pixels 440configured to sample focused optical image information and generate acorresponding electrical representation. Each pixel 440 samplesintensity information for locally incident light and stores theintensity information within associated analog sampling circuits. In oneembodiment, the intensity information comprises a color intensity valuefor each of a red, a green, and a blue color channel. Row logic 412includes logic circuits configured to drive row signals associated witheach row of pixels. The row signals may include, without limitation, areset signal, a row select signal, and at least two independent samplecontrol signals. One function of a row select signal is to enableswitches associated with analog sampling circuits within a row of pixelsto couple analog signal values (e.g., analog current values or analogvoltage values) to a corresponding column output signal, which transmitsthe analog signal value to column read out circuit 420. Column read outcircuit 420 may be configured to multiplex the column output signals toa smaller number of column sample signals, which are transmitted toanalog-to-digital unit 422. Column read out circuit 420 may multiplex anarbitrary ratio of column output signals to column sample signals.Analog-to-digital unit 422 quantizes the column sample signals fortransmission to interconnect 334 via input/output interface 426.

In one embodiment, the analog signal values comprise analog currents,and the analog-to-digital unit 422 is configured to convert an analogcurrent to a corresponding digital value. In other embodiments, columnread out circuit 420 is configured to convert analog current values tocorresponding analog voltage values (e.g. through a transimpedanceamplifier or TIA), and the analog-to-digital unit 422 is configured toconvert the analog voltage values to corresponding digital values. Incertain embodiments, column read out circuit 420 implements an analoggain function, which may be configured according to a digital gainvalue.

In one embodiment, control unit 414 is configured to generate detailedtiming control signals for coordinating operation of row logic 412,column read out circuit 420, analog-to-digital unit 422, input outputinterface unit 426, and statistics unit 416.

In one embodiment, statistics unit 416 is configured to monitor pixeldata generated by analog-to-digital unit 422 and, from the monitoredpixel data, generate specified image statistics. The image statisticsmay include, without limitation, histogram arrays for individual pixelcolor channels for an image, a histogram array for intensity valuesderived from each pixel intensity value for an image, intensity sumvalues for each color channel taken over an image, a median intensityvalue for an image, an exposure value (EV) for an image, and the like.Image statistics may further include, without limitation, a pixel countfor pixels meeting certain defined criteria, such as a pixel count forpixels brighter than a high threshold intensity, a pixel count forpixels darker than a low threshold intensity, a weighted pixel sum forpixels brighter than a high threshold intensity, a weighted pixel sumfor pixels darker than a low threshold intensity, or any combinationthereof. Image statistics may further include, without limitation, curvefitting parameters, such as least squares parameters, for linear fits,quadratic fits, non-quadratic polynomial fits, exponential fits,logarithmic fits, and the like.

Image statistics may further include, without limitation, one or moreparameters computed from one or more specified subsets of pixelinformation sampled from pixel array 410. One exemplary parameterdefines a subset of pixels to be a two-dimensional contiguous region ofpixels associated with a desired exposure point. Here, an exposureparameter may be computed, for example, as a median intensity value forthe region, or as a count of pixels exceeding a threshold brightness forthe region. For example, a rectangular region corresponding to anexposure point may be defined within an image associated with the pixelarray, and a median intensity may be generated for the rectangularregion, given certain exposure parameters such as exposure time and ISOsensitivity.

Image statistics may be accumulated and computed as digital samplesbecome available from pixel array 410. For example, image statistics maybe accumulated as digital samples are generated by the analog-to-digitalunit 422. In certain embodiments, the samples may be accumulated duringtransmission through interconnect 334. In one embodiment, the imagestatistics are mapped in a memory-mapped register space, which may beaccessed through interconnect 334. In other embodiments, the imagestatistics are transmitted in conjunction with transmitting pixel datafor a captured image. For example, the image statistics for a givenimage may be transmitted as in-line data, following transmission ofpixel intensity data for the image.

In one embodiment, image statistics are computed using a fixed-functionlogic circuit comprising statistics unit 416. In other embodiments,image statistics are computed via a programmable processor comprisingstatistics unit 416. In certain embodiments, programming instructionsmay be transmitted to the programmable processor via interconnect 334.

In one embodiment, control unit 414 is configured to adjust exposureparameters for pixel array 410 based on images statistics for a previousimage. In this way, image sensor 332 may advantageously determine properexposure parameters per one or more specified exposure points withoutburdening processor resources within processor complex 310, and withoutincurring concomitant latencies. The proper exposure parameters may bedetermined by sampling sequential images and adjusting the exposureparameters for each subsequent image based on exposure parameters for acorresponding previous image. The exposure parameters for a givencaptured image may be read by camera interface unit 386 and stored asmetadata for the image.

In one embodiment, input/output interface unit 426 is configured tomodify pixel intensity data associated with a captured frame based oncertain image statistics. In one implementation, input/output interfaceunit 426 adjusts white balance of an image during transmission of imagedata through interconnect 334. Red, green, and blue components of eachpixel may be scaled based on previously computed image statistics. Suchimage statistics may include a sum of red, green, and blue components.With these sums, input/output interface unit 426 may be configured toperform a conventional gray world white balance correction.Alternatively, the image statistics may include quadratic curve fitparameters. With quadratic fit components, input/output interface unit426 may be configured to perform a quadratic white balance mapping.Additional embodiments provide for illuminator identification viaselecting for pixels above a lower threshold and below an upperthreshold for consideration in determining white balance. Still furtherembodiments provide for color temperature identification by mappingselected samples to a color temperature snap-point. Mapping colortemperature to a snap-point thereby applies an assumption that sceneillumination is provided by an illuminator having a standard colortemperature. In each example, image statistics may be optionally appliedto adjust pixel information prior to transmission via interconnect 334.

In an alternative embodiment, statistics unit 416, as well as pixelmodification functions discussed herein with respect to input/outputinterface unit 426 are instead implemented within sensor interface 386,residing within processor complex 310. In such an embodiment, power andheat dissipation associated with statistics unit 416 and related pixelmodification functions is shifted away from pixel array 410, which mayincorporate circuitry that is sensitive to heat. In another alternativeembodiment, statistics unit 416, as well as pixel modification functionsdiscussed herein with respect to input/output interface unit 426 areinstead implemented within a separate die disposed within camera module330. In such an embodiment, related power and heat dissipation is alsoshifted away from pixel array 410. In this embodiment, camera module 330is configured to offer statistics and pixel modification functions inconjunction with a conventional processor complex 310, which may beconfigured to include a conventional sensor interface.

FIG. 4B illustrates a pixel array 410 within image sensor 332, accordingto one embodiment of the present invention. Pixel array 410 comprisesone or more pixels 440, and each pixel comprises one or more cells. Asshown, each pixel comprises cells 442, 443, 444, and 445. Each cell isconfigured to convert an incident light signal into a correspondingelectrical signal. The electrical signal may be integrated over timeduring an exposure period for each cell. An integrated electrical signalmay then be transmitted via column signals 432 to analog-to-digital unit422 by way of column read out circuit 420. Analog-to-digital unit 422converts integrated electrical signals to a corresponding digital valueas part of an overall process of digitizing an image captured withinpixel array 410. Row logic 412 generates row control signals 430,configured to provide reset, row selection, and exposure sample timecontrol for each corresponding row of pixels 440. For example, rowcontrol signals 430(0) may be configured to direct pixels 440(0)-440(a)to initialize state, capture an optical signal, and convert the opticalsignal to a set of corresponding integrated electrical signals residingwithin pixels 440(0)-440(a). Row control signals 430(0) then directpixels 440(0)-440(a) to drive corresponding integrated electricalsignals onto column signals 432(0)-432(c), availing at least a portionof the integrated electrical signals within a row of pixels 440 toanalog-to-digital unit 422 for conversion to corresponding digitalvalues.

FIG. 4C illustrates a color filter configuration comprising red, green,and blue filters for one pixel 440 of FIG. 4B within pixel array 410 ofFIG. 4A, according to one embodiment of the present invention. As shown,color filters for red, green, green, and blue are organized to interceptand filter incident light transmitted optically to corresponding cells.For example, cell 442 within pixel cell 440(0) of FIG. 4B receivesincident light filtered by a red (R) filter, cell 443 of cell 440(0)receives focused incident light filtered by a green (G) filter, and soforth.

FIG. 4D illustrates a color filter configuration comprising red, green,blue, and white filters for one pixel 440 within pixel array 410,according to one embodiment of the present invention. Here, a clear(white) filter is used to provide greater intensity sensitivity for onecell within pixel 440. For example, cell 445 of pixel 440(0) may receivewhite incident light, providing greater overall intensity than red,green, or blue filtered incident light, received by cells 442, 443, and444.

FIG. 4E illustrates a color filter configuration comprising cyan,magenta, yellow, and white filters for one pixel 440 within pixel array410, according to one embodiment of the present invention. Here, a cyan,magenta, yellow (CMY) color space is sensed by cells 442, 443, 444,while cell 445 senses intensity based a white sample of incident lightfor greater overall sensitivity.

FIG. 4F illustrates a cross-section of cells 460 within pixel array 410,according to one embodiment of the present invention. A lens array 467comprises an array of micro lenses 466, each fabricated to align with acorresponding photo detector/photo diode (PD) 462 residing within anassociated cell 460. In one embodiment cells 460(0) and 460(1)correspond to cells 442 and 443, respectively. A filter 464 determineswhich color of light a given color cell 460 receives for detection.Here, color refers to a range of wavelengths that generally correspondsto human color perception. Each micro lens 466 is configured toconcentrate incident light, from focused optical scene information, intoPD 462. By concentrating light from a larger surface area than that ofPD 462, greater sensitivity may be achieved.

FIG. 5 is a circuit diagram for a conventional photo-sensitive cell 500within a pixel, implemented using complementary-symmetry metal-oxidesemiconductor (CMOS) devices. Photo-sensitive cell 500 may be used toimplement cells comprising a conventional pixel. A photodiode (PD) 510is configured to convert incident light 512 into a photodiode current(I_PD). Field-effect transistors (FETs) 520, 522, 524, 526, andcapacitor C 528 are configured to integrate the photodiode current overan exposure time, to yield a resulting charge associated with capacitorC 528. Capacitor C 528 may comprise a distinct capacitor structure, aswell as gate capacitance associated with FET 524, and diffusion to wellcapacitance, such as drain capacitance, associated with FETS 520, 522.

FET 520 is configured to provide a path to charge node 529 to a voltageassociated with voltage supply V2 when reset0 530 is active (e.g., low).FET 522 provides a path for the photodiode current to discharge node 529in proportion to an intensity of incident light 512, thereby integratingincident light 512, when sample 534 is active (e.g., high). Theresulting charge associated with capacitor C 528 is an integratedelectrical signal that is proportional to the intensity of incidentlight 512 during the exposure time. The resulting charge provides avoltage potential associated with node 529 that is also proportional tothe intensity of incident light 512 during the exposure time.

When row select 536 is active (e.g., high), FET 526 provides a path foran output signal current from voltage source V1 through FET 524, to out538. FET 524 converts a voltage on node 529, into a corresponding outputcurrent signal through node out 538. During normal operation, incidentlight sampled for an exposure time corresponding to an active time forsample 534 is represented as a charge on capacitor C 528. This chargemay be coupled to output signal out 538 and read as a correspondingcurrent value. This circuit topology facilitates non-destructive readingof charge on node 529.

FIG. 6A is a circuit diagram for a photo-sensitive cell 600, accordingto one embodiment. An instance of photo-sensitive cell 600 may implementone cell of cells 442-445 comprising a pixel 440. As shown,photo-sensitive cell 600 comprises two analog sampling circuits 601, andphotodiode 620. Analog sampling circuit 601(A) comprises FETs 622, 624,626, 628, and node C 610. Analog sampling circuit 601(B) comprises FETs652, 654, 656, 658, and node C 640.

Node C 610 represents one node of a capacitor that includes gatecapacitance for FET 624 and diffusion capacitance for FETs 622 and 628.Node C 610 may also be coupled to additional circuit elements (notshown) such as, without limitation, a distinct capacitive structure,such as a metal-oxide stack, a poly capacitor, a trench capacitor, orany other technically feasible capacitor structures. Node C 640represents one node of a capacitor that includes gate capacitance forFET 654 and diffusion capacitance for FETs 652 and 658. Node C 640 mayalso be coupled to additional circuit elements (not shown) such as,without limitation, a distinct capacitive structure, such as ametal-oxide stack, a poly capacitor, a trench capacitor, or any othertechnically feasible capacitor structures.

When reset1 630 is active (low), FET 628 provides a path from voltagesource V2 to node C 610, causing node C 610 to charge to the potentialof V2. When sample1 632 is active, FET 622 provides a path for node C610 to discharge in proportion to a photodiode current (I_PD) generatedby photodiode 620 in response to incident light 621. In this way,photodiode current I_PD is integrated for a first exposure time whensample1 632 is active, resulting in a corresponding voltage on node C610. When row select 634 is active, FET 626 provides a path for a firstoutput current from V1 to output outA 612. The first output current isgenerated by FET 624 in response to the voltage on C 610. When rowselect 634 is active, the output current at outA 612 is thereforeproportional to the integrated intensity of incident light 621 duringthe first exposure time.

When reset2 660 is active (low), FET 658 provides a path from voltagesource V2 to node C 640, causing node C 640 to charge to the potentialof V2. When sample2 662 is active, FET 652 provides a path for node C640 to discharge according to a photodiode current (I_PD) generated byphotodiode 620 in response to incident light 621. In this way,photodiode current I_PD is integrated for a second exposure time whensample2 662 is active, resulting in a corresponding voltage on node C640. When row select 664 is active, FET 656 provides a path for a secondoutput current from V1 to output outB 642. The second output current isgenerated by FET 654 in response to the voltage on C 640. When rowselect 664 is active, the output current at outB 642 is thereforeproportional to the integrated intensity of incident light 621 duringthe second exposure time.

Photo-sensitive cell 600 includes independent reset signals reset1 630and reset2 660, independent sample signals sample1 632 and sample2 662,independent row select signals row select1 634 and row select2 664, andindependent output signals outA 612 and outB 642. In one embodiment,column signals 432 of FIG. 4B comprise independent signals outA 612 andoutB 642 for each cell within each pixel 440 within a row of pixels. Inone embodiment, row control signals 430 comprise signals for row select1634 and row select2 664, which are shared for a given row of pixels.

A given row of instances of photo-sensitive cell 600 may be selected todrive respective outA 612 signals through one set of column signals 432.The row of instances of photo-sensitive cell 600 may also be selected toindependently drive respective outB 642 signals through a second,parallel set of column signals 432. In one embodiment, reset1 630 iscoupled to reset2 660, and both are asserted together.

Summarizing the operation of photo-sensitive cell 600, two differentsamples of incident light 621 may be captured and stored independentlyon node C 610 and node C 640. An output current signal corresponding tothe first sample of the two different samples may be coupled to outputoutA 612 when row select1 634 is active. Similarly, an output currentsignal corresponding to the second of the two different samples may becoupled to output outB 642 when row select2 664 is active.

FIG. 6B is a circuit diagram for a photo-sensitive cell 602, accordingto one embodiment. An instance of photo-sensitive cell 602 may implementone cell of cells 442-445 comprising a pixel 440. Photo-sensitive cell602 operates substantially identically to photo-sensitive cell 600 ofFIG. 6A, with the exception of having a combined output signal out 613rather than independent output signals outA 612, outB 642. During normaloperation of photo-sensitive cell 602, only one of row select1 634 androw select2 664 should be driven active at any one time. In certainscenarios, photo-sensitive cell 602 may be designed to advantageouslyimplement cells requiring less layout area devoted to column signals 432than photo-sensitive cell 600.

FIG. 6C is a circuit diagram for a photo-sensitive cell 604, accordingto one embodiment. Photo-sensitive cell 604 operates substantiallyidentically to sensitive cell 600 of FIG. 6A, with the exception ofimplementing a combined row select 635 rather than independent rowselect signals row select1 634 and row select2 664. Photo-sensitive cell604 may be used to advantageously implement cells requiring less layoutarea devoted to row control signals 430.

Although photo-sensitive cell 600, photo-sensitive cell 602, andphoto-sensitive cell 604 are each shown to include two analog samplingcircuits 601, persons skilled in the art will recognize that thesecircuits can be configured to instead include an arbitrary number ofanalog sampling circuits 601, each able to generate an independentsample. Furthermore, layout area for a typical cell is dominated byphotodiode 620, and therefore adding additional analog sampling circuits601 to a photo-sensitive cell has a relatively modest marginal impact onlayout area.

In general, sample1 632 and sample2 662 may be asserted to an activestate independently. In certain embodiments, sample1 632 and sample2 662are asserted to an active state sequentially, with only one analogsampling circuit 601 sourcing current to the photodiode 620 at a time.In other embodiments, sample1 632 and sample2 662 are asserted to anactive state simultaneously to generate images that are sampledsubstantially concurrently, but with each having a different effectiveexposure time.

When both sample1 632 and sample2 662 are asserted simultaneously,photodiode current I_PD will be divided between discharging node C 610and node C 640. For example, if sample1 632 and sample2 662 are bothinitially asserted, then I_PD is split initially between dischargingnode C 610 and discharging node C 640, each at an initial dischargerate. A short time later, if sample2 662 is unasserted (set toinactive), then C 610 is discharged at a faster rate than the initialdischarge rate. In such a scenario, C 640 may be used to capture a colorcomponent of a pixel within a first image having a less sensitiveexposure (shorter effective exposure time), while C 610 may be used tocapture a corresponding color component of a pixel within a second imagehaving a more sensitive exposure (longer effective exposure time). Whileboth of the above color components were exposed according to differenteffective and actual exposure times, both color components were alsocaptured substantially coincidentally in time, reducing the likelihoodof any content change between the first image and the second image.

In one exemplary system, three substantially identical analog samplingcircuits 601 are instantiated within a photo-sensitive cell. In a firstsampling interval lasting one half of a unit of time, all three analogsampling currents are configured to source current (sample signalactive) into the photodiode 620, thereby splitting photodiode currentI_PD substantially equally three ways. In a second sampling interval,lasting one unit of time, a first of the three analog sampling circuits601 is configured to not continue sampling and therefore not sourcecurrent into the photodiode 620. In a third sampling interval, lastingtwo units of time, a second of the three analog sampling circuits 601 isconfigured to not continue sampling and therefore not source currentinto the photodiode 620.

In this example, the first analog sampling circuit 601 is able tointegrate one quarter of the photodiode current multiplied by time asthe second analog sampling circuit 601, which was able to integrate onequarter of the photodiode current multiplied by time as the third analogsampling circuit 601. The second analog sampling circuit 601 may beassociated with a proper exposure (0 EV), while the first analogsampling circuit 601 is therefore associated with a two-stop underexposure (−2 EV), and the third analog sampling circuit 601 is thereforeassociated with a two-stop over exposure (+2 EV). In one embodiment,digital photographic system 300 of FIG. 3C determines exposureparameters for proper exposure for a given scene, and subsequentlycauses the camera module 330 to sample three images based on theexposure parameters. A first image of the three images is sampledaccording to half an exposure time specified by the exposure parameters(−2 EV), a second image of three images is sampled according to theexposure time specified by the exposure parameters (0 EV), while a thirdimage of three images is sampled according to twice the exposure timespecified by the exposure parameters (2 EV). The first image is sampledconcurrently with the second image and third image, while the secondimage is sample concurrently with the third image. As a consequence ofconcurrent sampling, content differences among the three images aresignificantly reduced and advantageously bounded by differences inexposure time between images, such as images comprising an image stack.By contrast, prior art systems sample images sequentially rather thanconcurrently, thereby introducing greater opportunities for contentdifferences between each image.

These three exposure levels (−2, 0, +2 EV) for images comprising animage stack are suitable candidates for HDR blending techniques,including a variety of conventional and well-known techniques. Incertain embodiments, conventional techniques may be implemented todetermine exposure parameters, including a mid-range exposure time, fora given scene associated with a proper exposure (0 EV). Continuing theabove example, the first sampling interval would implement an exposuretime of half the mid-range exposure time. The second sampling intervalwould implement the mid-range exposure time, and the third samplinginterval would implement an exposure time of twice the mid-rangeexposure time.

In other embodiments, the analog sampling circuits 601 are notsubstantially identical. For example, one of the analog samplingcircuits 601 may include twice or one half the storage capacitance (suchas the capacitance associated with node C 610 of FIG. 6A) of a differentanalog sampling circuit 601 within the same pixel. Persons skilled inthe art will understand that relative sample times for each differentanalog sampling circuit 601 may be computed based on relativecapacitance and target exposure ratios among corresponding images.

In one embodiment, image sensor 332 comprising pixels 440 fabricated toinclude two or more instances of analog sampling circuits 601 isconfigured to sample one or more ambient image and sequentially sampleone or more images with strobe illumination.

FIG. 6D depicts exemplary physical layout for a pixel 440 comprisingfour photo-sensitive cells 442, 443, 444, 445, according to oneembodiment. As shown, each photo-sensitive cell 442, 443, 444, 445includes a photodiode 620 and analog sampling circuits 601. Two analogsampling circuits 601 are shown herein, however in other embodiments,three, four, or more analog sampling circuits 601 are included in eachphoto-sensitive cell.

In one embodiment, column signals 432 are routed vertically betweenphoto-sensitive cells 442 and 443, and between photo-sensitive cells 444and 445. Row control signals 430 are shown herein as running betweenphoto-sensitive cells 442 and 444, and between photo-sensitive cells 443and 445. In one embodiment, layout for cells 442, 443, 444, and 445 isreflected substantially symmetrically about an area centroid of pixel440. In other embodiments, layout for the cells 442, 443, 444, and 445is instantiated without reflection, or with different reflection thanshown here.

FIG. 7A illustrates exemplary timing for controlling cells within apixel array to sequentially capture an ambient image and a strobe imageilluminated by a strobe unit, according to one embodiment of the presentinvention. As shown, an active-low reset signal (RST) is asserted to anactive low state to initialize cells within the pixel array. Each cellmay implement two or more analog sampling circuits, such as analogsampling circuit 601 of FIGS. 6A-6C, coupled to a photodiode, such asphotodiode 620. In one embodiment, each cell comprises an instance ofphoto-sensitive cell 600. In another embodiment, each cell comprises aninstance of photo-sensitive cell 602. In yet another embodiment, eachcell comprises an instance of photo-sensitive cell 604. In still yetanother embodiment, each cell comprises an instance of a photo-sensitivecell that includes two or more technically feasible analog samplingcircuits, each configured to integrate a signal from a photodiode, storean integrated value, and drive a representation of the integrated valueto a sense wire, such as a column signal, such as a column signal 432.

A first sample enable signal (S1) enables a first analog samplingcircuit comprising a first analog storage plane to integrate a signalfrom an associated photodiode. A second sample enable signal (S2)enables a second analog sampling circuit comprising a second analogstorage place to integrate the signal from the photodiode. In oneembodiment, both reset1 630 and reset2 660 correspond to reset signalRST, sample1 632 corresponds to S1, and sample2 662 corresponds to S2.Furthermore, row select1 634 corresponds to RS1 and row select 664corresponds to RS2. In certain embodiments, RST is asserted brieflyduring each assertion of S1 and S2 to bias the photodiode prior tosampling the photodiode current. In certain other embodiments, eachphotodiode is coupled to a FET that is configured to provide a resetbias signal to the photodiode independent of the RST signal. Suchbiasing may be implemented in FIGS. 7B through 7F.

An Out signal depicts an analog signal being driven from an analogsampling circuit 601. The Out signal may represent outA 612, outB 642,or Out 613, depending on a particular selection of analog samplingcircuit 601. For example, in an embodiment that implementsphoto-sensitive cell 602, RS1 and RS2 are asserted mutually exclusivelyand the Out signal corresponds to Out 613.

A strobe enable signal (STEN) corresponds in time to when a strobe unitis enabled. In one embodiment, camera module generates STEN tocorrespond in time with S1 being de-asserted at the conclusion ofsampling an ambient image (Amb1).

FIG. 7B illustrates exemplary timing for controlling cells within apixel array to concurrently capture an ambient image and an imageilluminated by a strobe unit, according to one embodiment of the presentinvention. As shown, the active duration of STEN is shifted in timebetween two different sampling intervals for the same ambient image.This technique may result in charge sharing between each analog samplingcircuit and the photodiode. In this context, charge sharing wouldmanifest as inter-signal interference between a resulting ambient imageand a resulting strobe image. Removing the inter-signal interference mayattenuated in the ambient image and the strobe image using anytechnically feasible technique.

FIG. 7C illustrates exemplary timing for controlling cells within apixel array to concurrently capture two ambient images having differentexposures, according to one embodiment of the present invention. Asshown, S1 and S2 are asserted active at substantially the same time. Asa consequence, sampling of the two ambient images is initiatedconcurrently in time. In other embodiments, the strobe unit is enabledduring Amb1 and at least one of the images comprises a strobe imagerather than an ambient image.

FIG. 7D illustrates exemplary timing for controlling cells within apixel array to concurrently capture two ambient images having differentexposures, according to one embodiment of the present invention. Asshown, S2 is asserted after S1 shifting the sample time of the secondimage to be centered with that of the first image. In certain scenarios,centering the sample time may reduce content differences between the twoimages.

FIG. 7E illustrates exemplary timing for controlling cells within apixel array to concurrently capture four ambient images, each havingdifferent exposure times, according to one embodiment of the presentinvention. Each of the four ambient images corresponds to an independentanalog storage plane. In certain embodiments, a strobe unit is enabledand strobe images are captured rather than ambient images.

FIG. 7F illustrates exemplary timing for controlling cells within apixel array to concurrently capture three ambient images havingdifferent exposures and subsequently capture a strobe image, accordingto one embodiment of the present invention.

While row select signals (RS1, RS2) are shown in FIGS. 7A through 7F,different implementations may require different row selectionconfigurations. Such configurations are within the scope and spirit ofdifferent embodiments of the present invention.

While various embodiments have been described above with respect to adigital camera 202 and a mobile device 204, any device configured toperform the method 100 of FIG. 1A or method 102 of FIG. 1B is within thescope and spirit of the present invention. In certain embodiments, twoor more digital photographic systems implemented in respective devicesare configured to sample corresponding image stacks in mutual timesynchronization.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

We claim:
 1. A method for generating an image stack associated with aphotographic scene, comprising: initializing a pixel array configured toinclude a set of analog storage planes; enabling simultaneousintegration of the photographic scene for two or more analog storageplanes within the set of analog storage planes; enabling integration toproceed during a first sampling interval; disabling integration for atleast one analog storage plane; and enabling integration to proceedduring a second sampling interval.
 2. The method of claim 1, furthercomprising determining exposure parameters that include at least thefirst timing interval and the second timing interval.
 3. The method ofclaim 1, further comprising receiving a capture command.
 4. The methodof claim 1, wherein the set of analog storage planes includes two ormore analog storage planes.
 5. The method of claim 1, wherein eachanalog storage plane within the set of analog storage planes comprisesanalog sampling circuits corresponding to pixels within the pixel array.6. The method of claim 1, wherein each analog storage plane within theset of analog storage planes comprises analog sampling circuitsconfigured to store a charge that is proportional to a photodiodecurrent integrated over a sampling interval.
 7. The method of claim 1,wherein each analog storage plane within the set of analog storageplanes comprises analog sampling circuits is configured to share ananalog output signal and drive the analog output signal in response to acorresponding row select signal being asserted.
 8. The method of claim7, wherein the shared analog output signal comprises a column signalwithin the pixel array.
 9. A camera module, configured to generate animage stack associated with a photographic scene by performing the stepsof: initializing a pixel array configured to include a set of analogstorage planes; enabling simultaneous integration of the photographicscene for two or more analog storage planes within the set of analogstorage planes; enabling integration to proceed during a first samplinginterval; disabling integration for at least one analog storage plane;and enabling integration to proceed during a second sampling interval.10. The camera module of claim 9, further configured to perform the stepof determining exposure parameters that include at least the firsttiming interval and the second timing interval.
 11. The camera module ofclaim 9, further configured to perform the step of receiving a capturecommand.
 12. The camera module of claim 9, wherein the set of analogstorage planes includes two or more analog storage planes.
 13. Thecamera module of claim 9, wherein each analog storage plane within theset of analog storage planes comprises analog sampling circuitscorresponding to pixels within the pixel array.
 14. The camera module ofclaim 9, wherein each analog storage plane within the set of analogstorage planes comprises analog sampling circuits configured to store acharge that is proportional to a photodiode current integrated over asampling interval.
 15. The camera module of claim 9, wherein each analogstorage plane within the set of analog storage planes comprises analogsampling circuits is configured to share an analog output signal anddrive the analog output signal in response to a corresponding row selectsignal being asserted.
 16. The camera module of claim 16, wherein theshared analog output signal comprises a column signal within the pixelarray.
 18. A photographic system, comprising: A processing unit,configured to generate exposure parameters including a first samplinginterval and a second sampling interval; and A camera module, configuredto sample a photographic scene according to exposure parameters byperforming the step of: initializing a pixel array configured to includea set of analog storage planes; enabling simultaneous integration of thephotographic scene for two or more analog storage planes within the setof analog storage planes; enabling integration to proceed during a firstsampling interval; disabling integration for at least one analog storageplane; and enabling integration to proceed during a second samplinginterval, wherein the camera module is configured to transmit image datacorresponding to each analog storage plane to the processing unit.